Spencer Greenberg (Modern Wisdom guest) tested the ability of 15 astrologers to see if they could demonstrate genuine astrological skill. Spencer got the help of 6 astrologers to design this study: in each round, each astrologer gets LOTS of information about a real person (answers to 43 questions) along with 5 full astrological charts. They then predict which is the person's real natal chart (the other 4 are decoys).
Why this study design? One of the most fundamental claims of astrology is that a person's natal chart contains information about that person's life and character. If true, astrologers should be able to correctly choose a person's chart at a rate well above random guessing.
The study results are based on 152 astrologer participants with prior astrological experience who predicted they would do better than random guessing at the task. Each astrologer tries to match people to their correct chart 12 times. If they're guessing completely at random (they have no skill because astrology doesn't actually work), they might get about 24 questions right (on average).
These astrologers were quite confident in their ability to match people to charts. Those with the least experience believed (after they had completed participation) that they'd gotten 5 out of 12 right, and those with the most experience thought they'd gotten 10 out of 12 right.
So, how did astrologers do overall? If they'd gotten even 23% of questions right (slightly above the 20% of random guessing), the study would have come out in favor of astrology. But astrologers as a group performed indistinguishable from random guessing, getting 20% right.
Perhaps the less experienced astrologers were just dragging down the performance of the group? The researchers looked at how performance varied based on astrological experience. More experienced astrologers did not do better than less experienced ones despite being far more confident. The researchers offered a $1000 prize for anyone getting at least 11 out of 12. Unfortunately, despite more than half of the astrologers believing that they had gotten 6 or more questions right (after completing the task), in actual fact, not a single astrologer got more than 5 right.
But, even if they aren't getting the answers right, do they at least agree with each other on what the right answers are? No. Astrologers had very low agreement with each other on the chart for each person. If astrologers picked charts at random, they would agree with each other 20% of the time. In the study, even the most experienced astrologers only agreed 28% of the time.
In conclusion, despite believing they could do it, the 152 astrologers seemed to lack any ability to match people to their astrological charts. — h/t Spencer Greenberg